Learning Preference Relations for Information Retrieval

In this paper we investigate the problem of learning a preference relation i~om a given set of ranked documents. We show that the Bayes’s optimal decision function, when applied to learning a preference relation, may violate transitivity. This is undesirable for information retrieval, because it is in conflict with a document ranking based on the user’s preferences. To overcome this problem we present a vector space based method that performs a linear mapping from documeats to scalar utility values and thus guarantees transitivity. The learning of the relation between documeats is formulated as a classification problem on pairs of documents and is solved using the principle of structural risk minimization for good generalization. The approach is extended to polynomial utility functions by using the potential function method (the so called "kernel trick"), which allows to incorporate higher order correlations of features into the utility function at minimal computational costs. The resulting algorithm is tested on aa example with artificial data. The algorithm successfully learns the utility function underlying the training examples and shows good classification performance. Introduction The task of supervised learning in information retrieval (IR) is mostly based on the assumption that a given document is either relevant or non-relevant. This holds for example for Rocchio’s feedback algorithm (Saiton 1968) and for the binary independence model (Robertson 1977) which is based on a Bayesian approach. classification approach was adopted and as classifications were considered to be partitions on a set of objects this reduces to learning equivalence relations from examples. But there is also the view that the similarity of the documents to the query represents the importance of the documents (Salton 1989, p. 317), which in turn means that a user need implies some preference relation on the documents, hi (Bolhnann & Wong 1987) and (Wong, Yao, & BoUmann 1988) the idea was developed to learn a preference relation instead of an equivalence relation. The learning of preference relations reduces to a standard classification problem if pairs of objects are considered, because a binary relation can be viewed as a subset of the Cartesian product. (Wong, Yao, Bollmann 1988) successfully applied linear classification and perceptron learning to this problem. In this paper we consider the situation that there are more than two relevance levels and that there exist several documents with different relevance levels which all have the same description. We find that an ideal Bayesian approach leads to inconsistencies, namely to the violation of transitivity. To overcome this problem, an algorithm is developed which enforces transitivity by learning a linear mapping from document descriptions to scalar utility values based on training examples that consist of pairs of document descriptions and their preference relation. The learning procedure is based on the principle of structural risk minimization (SRM) (Vapnik 1995), which is known for its good generalization properties (for an application of SRM to document classification see (Joachims 1997)). The linear approach generalized to include nonlinear utility functions, which are able to capture correlations between the features, by applying the so-called "kernel-trick". The paper is structured as follows: First, the learning of preference relations is formulated as a classification problem on pairs of document descriptions and the inconsistency of the Bayesian approach is demonstrated. In the following, the linear vector space model is introduced and structural risk minimization is applied for learning the weight vector. Then, this approach is generalized to include nonlinear utility functions by applying the "kernel trick". Finally, we present some numerical experiments to demonstrate the validity of the approach. The Problem of Transitivity Let us consider a static document space denoted by D with documents d E D being represented by feature vectors d = (dl,d2,...,dn)’E :D where n denotes the number of the features dk. The user determines a preference relation on the documents used for training, and generates a training set S consisting of l pairs (d, d’) of document descriptions together with their relations d *> d~: From: AAAI Technical Report WS-98-05. Compilation copyright © 1998, AAAI (www.aaai.org). All rights reserved. no. of documents relevance levels d d’ ] d"